Robustness Tests and Statistical Inference

Robustness tests emerged as social scientists’ response to the uncertainty they face in specifying empirical models. We argue that the logic of robustness testing warrants a fundamental change in how researchers make inferences in their analysis of observational data. The dominant conception of robustness, which assesses whether the estimated effects remain statistically significant in all robustness test models, results in a flawed inferential logic. Even if the baseline estimation model were correctly specified, null hypothesis significance testing is problematic. It loses its inferential value when multiple models are estimated to explore the stability of the baseline model’s estimated effect to plausible alternative specification choices. We provide an operational definition of robustness as stability in effect size and show how, despite model uncertainty, robustness tests can improve the validity of inferences if such tests are embraced as an integral part of research design.